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Sir: 

The following amendments and remarks are submitted in response to the second non- 
final Office Action mailed on January 31, 2003, in connection with the above-identified 
application. 

Claims 1-20 are pending. 

Reconsideration of the application is respectfully requested for the following reasons. 

In the Office Action, the Examiner rejected claims 1-3 and 6 under 35 U.S.C. §103(a) for 
being obvious in view of a combination formed between the Ma and Liddy patents. Applicants 
traverse this rejection for the following reasons. 

Claim 1 recites a multimedia data structure which reflects a change of user relevance 
feedback for determining weights of image features used for an image search. The data structure 
includes (a) information describing at least one feature of a certain image, (b) recent user 
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feedback information based on user relevance feedback, and (c) whole feedback information 
based on die user relevance feedback. 

In order to establish a prima facie case of obviousness for claim 1, two reqioirements must 
be satisfied. First, the cited references must teach or suggest all the features in claim 1. Second, 
there must have been some teaching or suggestion in existence at the time the claimed invention 
was made that would have led one of ordinary skill in the art to combine the references in an 
attempt to form the invention. See MPEP §2143.01 and In re Rouffet, 47 USPQ.2d 1459 (Fed. 
Cir. 1997). 

The Ma patent discloses a method for retrieving images from a database. The method 
includes performing a first search based on a user-generated query indicative of an image of 
interest. Images are retrieved from the database and displayed based on one or more features 
in the query. The user views the displayed images and indicates whether each of them is relevant 
or irrelevant. The relevant images are grouped into a first cluster and the irrelevant images into 
a second cluster. A correlation matrix is constmcted to denote the relatedness (or lack thereof 
between the clusters) and weights may be assigned based on the correlation matrix. 

Subsequent searches are performed based on the generated clusters. For example, if an 
image is retrieved during a second search and indicated to be relevant, then the entire cluster to 
which the image belongs may be deemed relevant. On the other hand, a user may exclude images 
from the relevant cluster. The Ma method thus continuously updates its relevant cluster based 
on recent user relevance information. 
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Claim 1 is different from the Ma patent in at least one respect. Claim 1 recites a data 
structure which includes both recent user feedback information and whole feedback 
information . As the Examiner acknowledged in the Office Action, the Ma patent fails to teach 
or suggest a data structure having whole feedback information. More specifically, the Ma patent 
does not teach or suggest a data structure used for determining weights of image features of an 
image search which includes all user feedback information input up to a current point in time. 
(See, for example, the disclosure at page 11, linesl-3, of Applicants' specification). 

To make up for these deficiencies, the Liddy patent was cited. The Liddy patent discloses 
a method for updating a neural network for retrieving "documents" from a database. (At column 
5, lines 35-39, the Liddy patent discloses that these "documents" are HTML web pages.) To 
keep the neural network relevant, the neural network is periodically re-trained so that it is up-to- 
date. The manner in which the neural network is updated is disclosed as follows: 

Periodically, the agent server 22 retrains (or trains) the artificial neural network 
common to all agents under an agent leader in accordance with the training set 
of relevant documents (step 54). The interval between training session s may be 
a parameter set by the user. For example, the interval may be 15 minutes . When 
training is about to occur, if the user has not selected any relevant documents, the 
agent server 22 automatically performs relevance feedback at s tep 52 by 
considering the top X number of documents having the highest retrieval value as 
relevant and includes such documents in the training set by setting their relevancy 
bits in the database to "1". For example, X may be equal to 10, however other 
numbers may be used. If the user has selected less than X number of documents 
as relevant, the automatic relevant feedback may be performed to supplement the 
number of documents in the test set until X documents are present. (Emphasis 
added). 
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From the above, it is clear that the Liddy method performs a document search by initially 
training a neural network based on a set of training documents, and then periodically re-trains 
the neural network to reflect only the most recent user relevance feedback information. For 
example, once the neural network is initially trained, one or more document searches are 
performed. After 15 minutes has passed, the neural network is then re-trained , taking into 
consideration only the documents which a user has deemed relevant from searches performed 
over the last 15 minutes. From these disclosures, it is therefore clear that the neural network 
(which the Examiner appears to have called a data structure) disclosed in the Liddy patent is 
based only on recent user relevance information. The Liddy patent does not teach or suggest that 
its neural network is based on whole feedback information , e.g., all user feedback information 
input up to a current point in time. In fact, by discloses that its neural network is re-trained every 
period, the Liddy patent at least implicidy teaches away from these features of the invention. 

The Liddy patent also discloses that if the user has not selected any relevant documents 
in recent searches, an update of the neural network is performed any way. Thus, the neural 
network Liddy uses to perform web page searches is updated every period to reflect only the 
most current user relevance feedback information and this is the case under any circumstances. 

In view of the foregoing considerations, it is respectfully submitted that the Ma and Liddy 
patents both individually and collectively fail to teach or suggest a data structure having "whole 
feedback information based on the user relevance information" recited in claim 1. Absent a 
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teaching or suggestion of this features, it is respectfully subniitted that the first requirement of 
establishing a prima facie case of obviousness of claim 1 cannot be satisfied. 

The second requirement for establishing a prima facie case of obviousness is that some 
teaching or suggestion must have existed at the time the claimed invention was made to modify 
the Ma method so that any data structure it uses to perform an image search includes both 
recent user feedback information and whole feedback information. None of the references of 
record provide such a teaching or suggestion. As noted above, the Liddy method uses only 
recent user feedback information to re-train the neural network it uses to perform document 
searches. Neither the Liddy patent nor the Ma patent itself teaches or suggests modifying a data 
structure which includes recent user feedback information for purposes of determining weight 
of image features to be used in an image search to also include whole feedback information to 
be used as a separate basis for determining these weights. 

Because neither requirement for establishing a prima facie case of obviousness of claim 
1 has been satisfied, it is respectfully submitted that claim 1 and its dependent claim are 
allowable over a Ma-Liddy combination. 

The Examiner rejected claims 4 and 14-20 under 35 USC § 103(a) for being obvious in 
view of a combination formed among the Ma, Liddy, and Cohen patents. Applicants traverse 
this rejection for the following reasons. 
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Claim 4 depends from claim 1. In order to render claim 4 obvious, the Cohen patent 
must therefore teach or suggest the features of claim 1 missing from the Ma and Liddy patents. 
The Cohen patent discloses a browser for performing a ke)rword search of internet web sites. 
In performing this search, a matcher compares a parameter of the retrieved website address 
(URL) with a keyword of the search and provides a score of the relevancy therebetween. 

The Cohen patent does not disclose including both recent user feedback information and 
whole feedback information in the same data structure , as recited in claim 1. Moreover, Cohen 
does not provide a teaching or suggestion for motivating one of ordinary skill in the art to 
modify a data structure including recent user feedback information to also include whole 
feedback information, so that, for example, weights of image features for performing image 
searches may be performed based on both types of information. 

Because the Cohen patent does not provide a teaching or suggestion sufficient to satisfy 
either requirement for establishing a prima facie case of obviousness of claim 1 when combined 
with the Ma and Liddy patents, it is respectfully submitted that claim 1 is allowable over this 
combination. It is therefore submitted that claim 4 is allowable over this combination by virtue 
of its dependency from claim 1. 

Applicants fiirther submit that claim 4 is allowable based on the features separately 
recited therein. Claim 4 recites that the data structiare of claim 1 further includes reliability 
information indicating the reliability of the recent user feedback information and 
reliability information indicating the reliability of the whole feedback information. The 
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Ma, Liddy, and Cohen patents fail to teach or suggest storing both feedback information and 
reliability information in the same data structure. 

More specifically, the Cohen patent does not teach or suggest a data structure having 
both recent user feedback information and reliability information indicating how reliable the 
recent user feedback information is. To provide these features, the Examiner pointed to column 
7, lines 51-56, of the Cohen patent. These lines of Cohen disclose providing a "score for a 
retrieved message, which reflects the relevancy and reliability of the message." These disclosures 
of Cohen make clear that a relevance score is assigned to the results of a keyword search 
performed by a web browser. Unlike the claimed invention, this score is not stored in a data 
structure which includes recent user feedback information used for determining weights of 
image features for an image search. Moreover, Cohen does not teach or suggest including both 
recent user feedback information and whole feedback information in the same data structure. 

The Cohen patent also does not teach or suggest a data structure having both whole 
feedback information and reliability information indicating how reliable the whole feedback 
information is. To provide these features, the Examiner pointed to column 2, lines 45-64, of the 
Cohen patent. These disclosures of Cohen merely indicate that a score is generated for each 
result (website URL) of a web browser search, and that this score may be adjusted to determine 
how relevant the result is compared with a keyword used to perform the search. The Cohen 
patent does not disclose storing whole feedback information in the same data stmcture as 
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reliability information used to store information indicative of how reliable the whole feedback 
information is. 

For at least the foregoing reasons, it is respectfully submitted that claim 4 is allowable 
over a Ma-Liddy-Cohen combination. 

Claim 14 recites a method of determining weights of image features used for an image 
search based on user relevance feedback. This method includes, in step (a), providing a 
multimedia data structure including information describing features of a certain image, recent 
user feedback information for the image and whole feedback information for the imag e, and 
reliability information corresponding to the recent user feedback information and whole 
feedback information . In step (b), the method updates the recent user feedback information and 
whole feedback information and their reliabilities by learning them in response to the user 
relevance feedback. In step (c), the method determines weights of image features in proportion 
to the reUabiUries of the recent feedback information, the whole feedback information, or both 
the recent feedback information and the whole feedback information . None of these features 
are taught or suggested by the Ma, Liddy, and Cohen patents. 

Claims 1 6 and 1 7 both recite determining reliability of recent user feedback information. 
None of the references of record teach or suggest determining reliability for this type of 
information. Claim 1 6 further recites determining this reliability in proportion to the consistency 
of a reccndy used pattern or feedback . None of these features are taught or suggested by the 
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references of record. Claim 17 further recites determining the reliability of the whole feedback 
information in proportion to the number of feedback concerned in learning . 

Claim 18 recites that the reliability of the whole feedback information is responsive to 
recorded user usage records wherein the recorded user usage records provide feedback to the 
reliability of the whole feedback information without user interaction. None of these features 
are taught or suggested by the references of record. 

Claim 19 recites a multimedia data stmcture reflecting change of a user relevance 
feedback for determining weights of image features used for an image search. This structure 
includes information describing at least one feature of a certain image, recent user feedback 
information based on user relevance feedback, whole feedback information based on the user 
relevance feedback, and reliability information indicating reliability of at least one of the user 
feedback information and whole feedback information. None of the references of record teach 
or suggest including both recent user feedback information and whole feedback information in 
the same data structure , nor do they teach or suggest reliability information of the type recited 
in claim 19. 

Claim 20 recites that the reliability information indicates reliability of both of the user 
feedback information and the whole feedback information. None of the references of record 
teach or suggest these features. 

Reconsideration and withdrawal of all the rejections and objections made by the 
Examiner is hereby respectfully requested. 
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In view of the foregoing amendments and remarks, it is respectfully submitted that the 
application is in condition for allowance. Favorable consideration and prompt allowance of the 
application is respectfully requested. 

Should the Examiner believe that further amendments are necessary to place the 
application in condition for allowance, or if the Examiner believes that a personal interview 
would be advantageous in order to more expeditiously resolve any remaining issues, the 
Examiner is invited to contact Applicants' undersigned attorney at the telephone number listed 
below. 

To the extent necessary, Applicants petition for an extension of time under 37 CFR 
§1.136. Please charge any shortage in fees due in connection with this application, including 
extension of time fees, to Deposit Account No. 16-0607 (Attorney Docket No. HI-0019) and 
credit any excess fees to the same Deposit Account. 



FLESHNER & KIM, LLP 
P.O. Box 221200 
Chantilly,VA 20153-1200 
Telephone No: (703) 502-9440 
Facsimile No: (703) 502-9596 




Carl R. Wesolowski 
Registration No. 40,372 
Samuel W. Ntiros 
Registration No. 39,318 
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